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The results for the Debye and Meissner screening masses of the gluons and the photon in the 
case of neutral and /3-equilibrated dense two-flavor quark matter are presented. In the limits of the 
normal phase and the ideal two-flavor color superconducting phase, the screening masses coincide 
with the known results. Most interestingly, we find that the Meissner screening masses squared can 
be negative, indicating a plasma-type chromomagnetic instability in dense quark matter. 
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I. INTRODUCTION 

It is known that sufficiently cold and dense baryonic 
matter is a color superconductor. It is possible that such 
a state of matter may naturally exist in the Universe in- 
side central regions of compact stars. For this reason, 
the topic of color superconductivity stirred a lot of in- 
terest in recent years Q, |^ |^ Q . (For reviews on color 
superconductivity see, for example, Ref. fsj.) 

Matter in the bulk of stars is electrically neutral and (3 
equilibrated. This means that the chemical potentials of 
different quarks satisfy some nontrivial relations. Such 
relations affect the pairing dynamics between quarks, 
and the ground state of matter is modified. It was ar- 
gued in Refs. 0, 0]) for example, that a mixture of the 
two-flavor color superconducting (2SC) phase and un- 
paired strange quarks is less favorable than the color- 
flavor-locked (CFL) phase when the charge neutrality is 
enforced. 

When the constituent mass of the strange quark is 
large, neutral two-flavor quark matter in (3 equilibrium 
can have a ground state called the gapless color super- 
conductor (g2SC) While the symmetry of the g2SC 
ground state is the same as that of the conventional two- 
flavor color superconductor (2SC), the spectrum of the 
fermionic quasiparticles is different. 

The existence of stable gapless color superconducting 
phases was confirmed in Refs. lo lllOl lO , and generalized 
to finite temperatures in Refs. fla . [lj| • It was also 
shown that a gapless color-flavor-locked (gCFL) phase 
can appear in neutral strange quark matter |l5l Il6j . 
A nonrelativistic analogue of gapless superconducting 
phases could a ppe ar in trapped gases of cold fermionic 
atoms 0, 0, Il9l l20l | . (Note that a mixed phase can 
be an alternative ground state in atomic [23| as well as 
quark [23, |U I3 systems provided the surface tension is 
small.) 

In this paper, we report on the gluon screening proper- 
ties in neutral dense two-flavor quark matter. Our results 
cover the gapped as well as the gapless 2SC phases. In 



the case of the ideal 2SC phase (i.e., without a mismatch 
between the Fermi momenta of different quarks), our re- 
sults reproduce the flndings of Refs. |2^|2y|. The most 
important result of this paper is the finding of a chromo- 
magnetic type plasma instability in neutral dense quark 
matter. We show that there exist five unstable gluon 
modes in the g2SC phase, and four of them persist even 
in the gapped phase when the mismatch is larger than 
a certain critical value. We argue that this instability 
may lead to a gluon condensation in dense quark mat- 
ter. Specific details of the condensate still remain to be 
clarified. 



II. GLUON SELF-ENERGY 

In the ground state of a two-flavor color superconduc- 
tor, the gauge symmetry group SU(3)c x U(l)cm is broken 
down to SU(2)c x U(l)om by the Anderson-Higgs me- 
chanism. Normally, this should lead to a generation of 
masses for the 5 gluons that correspond to the broken 
generators. This would be a standard description of the 
Meissner effect. As we shall see below, this is not always 
the case in dense quark matter. 

In order to study the gluon screening properties in the 
system at hand, we follow an approach similar to that 
in Refs. 25, 26]. The polarization tensor in momentum 
space has the following general structure: 



(2^) 



rTr, 



f^5(/0f^5(/^-F) 



(1) 

The trace here runs over the Nambu-Gorkov, flavor, color 
and Dirac indices. The 4- momenta are denoted by capital 
letters, e.g., P — {po,p)- The explicit form of vertices 
is 



^ A — 



diag(5 7^T^,-5 7''Tj) for A-l,...,8, 
diag(e 7^g, -ejf'Q) for A = 9 , 



(2) 
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where Ta and Q are the generators of the color and the 
electromagnetic gauge transformations (with the A = 9 
gauge boson corresponding to the photon). Note that the 
strong and electromagnetic coupling constants, g and e. 
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are included in the definition of the vertices. The inverse 
of the quark propagator is defined as 



[siP)Y 



A+ 



A- 



(3) 



with the off-diagonal elements A = ~ie''e'-f^A and 
A+ = 7° (A-)^7'' = -ie''e7^A*. Here A is the diquark 
gap parameter, while (e^)^^ = e''"'' and (e)*-' = e*^ are 
antisymmetric tensors in the color and flavor spaces, re- 
spectively. Without losing generality, we assume that the 
quarks are massless in dense quark matter jji, .27.] . Then 
the free quark propagators Gq (P) read 

[G^] = 7"(Po - P ± A)A+ -t- 7°(Po + P ± A) A;, (4) 

where = i [l ± 7^(7 • p)/p\ are the positive and ne- 
gative energy projectors, and fi is the matrix of chemical 
potentials in the color and flavor space. In5 equilibrium, 
the nontrivial elements of matrix fi read [3 Il2l | 



of an intermediate strength of the coupling, the ground 
state is given by the gapless 2SC phase (8i.il2l|. 

In order to describe the most general situation, we 
parametrize all the results below with the dimensionless 
ratio A/(5/i, where Sfi = fie/'2 is the mismatch between 
the Fermi momenta of paired quarks which is determined 
by the neutrality and the /3-equilibrium conditions. The 
value A/Sfi = corresponds to the normal phase, while 
A/S^ < 1 and A/S^ > 1 correspond to the gapless and 
the gapped 2SC phases, respectively [sLIl^. 

Screening masses of the gluons with A = 1,2,3. The 
general expression for the polarization tensor n^'^(0,p) 
with A^B = 1,2,3 is diagonal. Then, by making use of 
the definition in Eq. (|UJ) , we arrive at the following result 
for the threefold degenerate Debye mass: 



m 



D,l 



2a, [(m-)^ + Sfi 

TT ^(^/i)2 - A2 



9{Sf^-A) 



ny/iSfiy - A^ 



0(6(1 ~ A), 



(11) 



Mur — fJ-ug — M ^ gMe + gMSj 



2 1 

+ 3' 

1 1 

+ 3' 

2 2 



1 2 



(5) 
(6) 
(7) 
(8) 



with /i, /ig, and /ig being the chemical potentials of the 
quark number, the electrical charge and the color charge, 
respectively. 



III. SCREENING MASSES OF GAUGE BOSONS 



The Debye masses mfj ^ and the Meissner masses 



TO 



M 



^ of gauge bosons are defined in terms of the eigen- 



values of the polarization tensor |25l l26l | . In the basis in 



which n^'^(0,p) is diagonal, they become 



'■D,A 



= -limn^'^^(0,p) 



= — — lim 



(9) 



2 p^o 



9tj 



PiPj 



n^;^(o,p). (10) 



Below, we give only the final results for these quantities. 
The details of the calculation will be presented elsewhere 

M 

As was shown in Refs. the ground state of neu- 

tral dense quark matter is determined by the strength 
of the diquark coupling constant. At weak diquark cou- 
pling, the Cooper pairing is in conflict with the constraint 
of charge neutrality. As a result, the ground state corre- 
sponds to the normal phase. At strong diquark coupling, 
on the other hand, the ground state is in the gapped 2SC 
phase, and the neutrality plays little effect. In the regime 



where a, = 



g^/47r, fi = {fiur + fJ'dg)/'2, and fj.^ 
A^ are the values of the "effective" Fermi 



momenta of the gapless quasiparticles [1, . The mass 
as a function of A/5fi is shown in the upper panel of 
Fig. n with a red solid line. It is easy to check that the 
result in Eq. 1)1111 reduces to the known expressions in the 
normal phase (i.e., A/Sfi — Q) l29l I3(tI| and in the ideal 
2SC phase (i.e., A/S^i = oo) 

We note that the Debye screening mass in Eq. Hll|l 
vanishes in the gapped phase (i.e., A/^/i > 1). As in the 
case of the ideal 2SC phase, this reflects the fact that 
there are no gapless quasiparticles charged with respect 
to the unbroken SU(2)c gauge group. In the gapless 2SC 
phase, such quasiparticles exist and the value of the De- 
bye screening mass is proportional to the density of states 
at the corresponding "effective" Fermi surfaces .12., 3Xj . 

In the limit when the ratio A/5/i approaches 1 from 
below (i.e., from the gapless phase), the formal value of 
the Debye mass in Eq. pi(l goes to infinity. This is be- 
cause the density of the gapless states goes to infinity 
at A/Sfi = 1 when the gapless quasiparticle dispersion 
relation becomes quadratic, i.e., ~ [p ~ fl)^/2A in 
notation of Ref. . 

The Meissner screening mass is defined by Eq. H10|l . 
An explicit calculation for the gluons of the unbroken 
SU(2)c shows that their Meissner masses are vanishing 
in the gapped and gapless 2SC phases. 



= 0. 



(12) 



Of course, this is in agreement with the general group- 
theoretical arguments. 

Screening masses of the gluons with A — 4,5, 6, 7. The 
polarization tensor n^'^(0,p) with A, B = 4, 5, 6, 7 is not 
diagonal. It becomes diagonal after switching to the 
physical basis in terms of A'^^ = (A4 ± iA^)/\/2 and 
A'^^ = (A^ ± iAff)/V2 fields. In the static hmit, all four 
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FIG. 1: Squared values of the gluon Debye (upper panel) 
and Meissner (lower panel) screening masses, devided by 
rUg = 4Q!s/i^/37r, as functions of the dimensionless parameter 
A/Sjj,. The red solid line denotes the results for the gluons 
with A = 1, 2, 3, the green long-dashed line denotes the results 
for the gluons with A = 4,5, 6, 7, and the blue short-dashed 
line denotes the results for the gluon with A = 8. 



eigenvalues of the polarization tensor are degenerate. By 
making use of the definition in Eq. we derive the 
following result for the corresponding Debye masses: 



2A2 



9{S^l - A) 



(13) 

Here we assumed that is vanishing which is a good 
app roximation in neutral two-flavor quark matter [T3 . 
1321 . The c omp lete result with nonzero /ig will be reported 
elsewhere 12^ . 

Now, the fourfold degenerate Meissner screening mass 
of the gluons with A — A, b, 6, 7 reads 



Tom 4 



37r 



A2 - 25^2 g^^^ 



2A2 



A2 



-0{Sfi - A) 



(14) 

Both results in Eqs. ((T^ and ((Tl|l interpolate between 
the known results in the normal phase (i.e., A/Su = 0) 
and in the ideal 2SC phase (i.e., A/Sfi = oo) [23. The 
screening masses are plotted in Fig. ^ using green long- 
dashed lines. 



The most interesting observation about the gluon 
screening properties here is that the expression on the 
right hand side of Eq. H14|l is negative when < A/Sfj, < 
\f2. The standard interpretation of such a result is 
the existence of a plasma type instability in the system 
HE 1^ HE HEl' In the case at hand, we find that the 
instability appears in the whole family of gapless 2SC 
phases (with < A/5/i < 1) and even in some gapped 
2SC phases (with 1 < A/(5/i < \/2). 

While one may argue that the plasma instability in 
the gapless phase is related to the instability discussed 
in Rcf. 37] , it is obviously not the case in the gapped 2SC 
phase when 1 < A /J^ < a/2. The arguments of Ref . [s^ 
are based on the fact that the ground state corresponds to 
a local maximum of the effective potential in the system 
with a fixed mismatch parameter, bp, = const. However, 
without imposing the charge neutrality {b\x = const), 
the gapped 2SC ground state with A/Sfj, £ (1, 1/2) corre- 
sponds to a local minimum of the effective potential. Al- 
though it is not the global minimum for A/Sfi € (1, \/2), 
it is unlikely that the gluon screening masses could probe 
the global structure of the effective potential. Thus, the 
true origin of the gluon instability is yet to be clarified. 

Screening masses and mixing of the 8th gluon and the 
photon. From the general arguments, one knows that 
there is a new U(l)em symmetry in the 2SC/g2SC phase. 
The new medium photon is a mixture of the 8th gluon 
and the vacuum photon (the corresponding generator is 

It is important to emphasize that the explicit result 
for the 00-components of the polarization tensor of the 
8th gluon and the vacuum photon have no mixing at 
Po = and p ^ 0. This is similar to the ideal 2SC 
case considered in Ref. |2^. (In view of this, one should 
be careful when interpreting the results for the Debye 
screening masses in a different basis of gauge fields |3q.) 
The expressions for the Debye screening masses read 



37r 



ZS^JL e{Sn - A) 



(15) 



(16) 



where a = e'^ /Air is the fine structure constant. 

In order to obtain the Meissner screening masses, we 
first derive all the components of the polarization tensor 
that span the space of the 8th gluon and the vacuum 
photon. At po = and p — » 0, the corresponding nonzero 
components, denoted as m\i in analogy with Eq. I|10|l . 
read 




S^i d{Sn - A) 
' ^{5p)^~A\ 

6n e{5pL-A) \ 



(17) 
(18) 
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Sj^_0i5^\ (19) 



of the fields, 



''M,87 



. The mixing disappears in the basis 



Alcos if + AI sirup, 
AZ = Al cos ip-Al sin ^, 



(20) 
(21) 



where the mixing angle ip is determined by the symmetry 
arguments, 



sm ip 



3a,, 



(22) 



As one might have expected, the mixing angle is the same 
in the gapless and the gapped phases. Of course, it also 
coincides with the mixing angle in the ideal 2SC case in 
Ref. J6J. 

It should be noted that the absence of a mixing be- 
tween the electrical modes of the 8th gluon and the vac- 
uum photon in the static limit, see Eqs. (|15|l and (|16|l . is 
not in conflict with the gauge invariance of the 2SC/g2SC 
ground state with respect to U(l)om- The two propagat- 
ing modes of the low-energy photon 7 of U(l)em should 
have transverse polarizations and, therefore, should come 
from the magnetic sector. The third, electrical mode of 
7 is not massless. It decouples from the low-energy the- 
ory and its presence is irrelevant for the gauge invariance 
with respect to U(l)om. 

The Meissner screening mass for the new gluon field is 



M,8 



4(3as -I- a)/i^ 
27^? 



1 - 



Sfi 0{6n - A) 



(23) 



which is shown in the lower panel of Fig. fusing a blue 
short-dashed line. The Meissner screening mass for the 
new photon 7 is vanishing, = 0. This is consistent 

with the absence of the Meissner effect for the unbroken 

U(l)cm. 

As is easy to see from Eq. (I23II . the medium modified 
8th gluon has a magnetic plasma instability in the gapless 
2SC phase. This is because the Meissner screening mass 
squared is negative when < A/Sfi < 1. 



IV. DISCUSSION 

In this paper, we calculated the Debye and Meissner 
screening masses of the gluons and the photon in the case 
of neutral, /3-equilibrated dense two-flavor quark matter 
(see Fig. Our results interpolate between the known 
values in the normal phase 29, 30] and in the ideal 2SC 
phase 0, 01 of quark matter. 

The Meissner screening properties of dense matter are 
most interesting. We find that there is a chromomag- 
netic plasma type instability in quark matter [39j |. This 
is driven by 5 unstable gluon modes {A = 4, 5, 6, 7, 8) in 



the g2SC phase (0 < A/S^ < 1): and by 4 modes (A = 

4, 5, 6, 7) in the gapped 2SC phase when 1 < A/Sfj, < \/2. 
One could connect the instability from the 8th gluon to 
a large density of gapless states in the g2SC phase. This 
might be related to the mechanism proposed in Ref. |37| . 
The other 4 unstable modes have no obvious connection 
with the existence of the g2SC phase because they lead 
to an instability even in the gapped phase. 

It is natural to assume that the instability, indicated 
by the negative values of the Meissner screening masses 
squared, will resolve in some type of a gluon condensa- 
tion. Indeed, the result m^j^ < suggests that the 
system stays in a false vacuum that corresponds to a 
local maximum of the effective potential for the gluon 
field. Then, the true vacuum is most likely given by the 
global minimum of the potential that corresponds to a 
nonzero expectation value of a gluon field, i.e., (^J^) 7^ 
for A e (4,5,6,7) or for A = 8, depending on which 
instability develops first. 

We cannot exclude the possibility that the true ground 
state has a condensate that breaks the rotational symme- 
try of the system. In fact, this would be the most natural 
outcome of a gluon condensation because it is the mag- 
netic components of gluons Af that drive the instability. 
One might even speculate that the mechanism is sim- 
ilar to that in Ref. [ilf. 

In passing, we note that the gluon type instability, indi- 
cated by negative values of the Meissner screening masses 
squared, is not directly related to the so-called Sarma in- 
stability ^2 ■ As was shown in Ref. , the Sarma insta- 
bility in the effective potential for the order parameter is 
removed when the neutrality condition is imposed. 

The nature of the instability observed here may re- 
semble the instability in anisotropic models of Refs. |U 

0, |3^ used for describing the initial stage of heavy 
ion collisions. However, the origins of the two instabil- 
ities are very different. In contrast to the situation in 
Refs. [33, I33. 135I l3^ . the quark distribution functions 
are completely isotropic in momentum space in neutral 
dense quark matter. 

In the future, it would be very interesting to investi- 
gate whether a similar instability develops in the gCFL 
phase lillll, where the low-energy quasiparticle spec- 
trum resembles the spectrum in the g2SC phase. 
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